The Influence of Luminance on Local Tone Mapping

We study the influence of the choice of color space for local tone mapping methods. Many local tone mapping methods do not perform well when applied independently to the three color channels of an RGB image. A common solution is to only treat the luminance channel. However, the question of which color space provides the most suitable luminance definition has not been addressed. The correlation between luminance and chrominance is known to have an influence on the rendered image but the relation between a measure of correlation and the appearance of the image has not yet been found. We consider four color transforms and introduce a measure to evaluate how well they decorrelate luminance and chrominance information. We apply two local tone mapping algorithms to the luminance channel given by the four transforms and visually compare the results. As each transform leads to another luminance definition, the resulting color images will be different as well. Our results confirm that less correlation between luminance and chrominance results in better performance of the local tone mapping algorithms. Namely, they provide a better increase in local contrast in the luminance channel and less hue shifts. However, we show that a perfect decorrelation is not always necessary. Introduction Tone mapping methods are a critical step in the reproduction of images. These methods can be classified into two groups: global and local tone mapping. Global methods treat the image as a whole using a mapping function. One input value is mapped to one and only one output value, which depends on the mapping function that can be image dependent [1]. Local methods treat the image spatially using local operators. One input value can be mapped to different output values depending on the surrounding pixel values. Global methods provide satisfying results for most of the captured images. Nevertheless, when the dynamic range of the captured scene greatly exceeds the dynamic range of the display, a local tone mapping is necessary to render pleasing images. Most local tone mapping algorithms are inspired by the Retinex theory of color vision [2]. Retinex aims to predict the sensation of color by making spatial comparisons of color surfaces across the image. In its first iteration, Retinex was applied independently to all three R,G,B channels. While treating R,G,B independently provides good results with global tone mapping methods, it becomes problematic for local tone mapping algorithms. Indeed, when applied locally such algorithms may create artifacts such as local graying out, hue shifts or color fringes, as illustrated in Figure 1. The left image of Figure 1 was obtained by applying the Multi-scale Retinex algorithm [3,4] to all three R,G,B channels independently, which tends to gray out the image. The right image was processed similarly with the Retinex-based adaptive filter algorithm [5,6]. Processing R,G,B independently causes a hue shift in this case. A well-accepted solution to avoid these artifacts is to treat the luminance independently from the chrominance [5,6,7,8,9,10]. However, none of these methods investigate the influence of the chosen color transform on the appearance of the treated image. Figure 1. Left: multi-scale Retinex applied to all R,G,B channels grays out the image. Right: The Retinex-based adaptive filter method applied to all R,G,B channels causes a hue shift. In this article, we investigate the influence of the color space transformation in the case of luminance-based local tone mapping methods. In particular, we focus on surround-based Retinex methods, which compute new pixel values (Ψnew) by taking the difference in the log domain between each pixel value and a weighted average of its surround: )) , ( log( )) , ( log( ) , ( y x mask y x y x new − Ψ = Ψ , (1) where Ψ is the luminance image. It is computed by a color transform applied to the input image, which is linear with respect to scene radiance. mask is the weighted average of pixel values in the surrounding of coordinate (x,y). Our aim is to relate a measure of the correlation between luminance and chrominance with the color rendition of images treated by surround-based Retinex methods. We consider four color transforms and define a measure to evaluate how well they decorrelate luminance and chrominance. Then, we test two Retinex-based local tone mapping algorithms with the four different color transforms and relate the results with our measure. We show that there is a relation between the visual representation of the rendered image and the measure of correlation. Color artifacts become visible when luminance and chrominance are significantly correlated. This article is structured as follows: Section 2 reviews background work about color rendition in the case of local tone mapping algorithms. Section 3 presents our measure and the four color transforms that we consider. Section 4 presents the two algorithms used for the test. Section 5 comments the images obtained with the two algorithms and the different color transforms. Conclusion and future work are given in section 6. Background Current methods provide solutions to overcome artifacts created by local tone mapping. Rahman et al. [3,4] discussed the graying out effect of surround-based Retinex algorithms and added a color restoration step to their Multi-Scale Retinex (MSR) algorithm. The Multi-Scale Retinex with Color Restoration (MSRCR) was studied by Funt and Barnard [7,11]. They concluded that the color restoration step compensates for the graying out effect by increasing the saturation, but has an unpredictable effect on the hue of the image. This graying out effect is due to the regional violations of the gray-world assumption intrinsic to the Retinex theory. Funt and Barnard thus suggest applying MSR to the luminance channel only. The treated luminance is then combined with the chrominance to obtain the final color image. They define the luminance as the average of the three color channels R,G,B. With this definition of luminance, some chromatic information remains in the luminance and vice-versa, which may lead to artifacts. In a recent article [6], we presented a Retinex-based method that applies an adaptive filter to the luminance channel. The luminance is defined by a principal component analysis (PCA) computed over an RGB input image. The use of a PCA is motivated by the fact that it has properties that intrinsically lead to an opponent representation of colors, which makes it biologically plausible [12,13]. The first component is all positive and has the largest share of signal energy. It represents the achromatic channel, carrying luminance information. The second and third components are opponent and represent the chrominance information. Moreover, PCA provides an optimal decorrelation of the three color channels. Fairchild and Johnson [8] developed a color appearance model (iCAM), which applies a local treatment to the luminance channel. The first stage of iCAM accounts for chromatic adaptation. Then, the image is transformed into an opponent representation. Only the luminance channel is processed to avoid the desaturation caused by the local treatment. Sobol [10] also applies his Retinex-based algorithm to the luminance channel only. Unlike previously mentioned methods that define the luminance as weighted sum of R,G,B color channels, his luminance definition is given by the maximum between these three channels. The final color image is obtained by adding the new luminance to the log-encoded RGB image. Thus, many local tone mapping methods first transform the input image into a luminance/chrominance representation and treat the luminance only: { } { }) , , ( , , 2 1 B G R f C C cs = Ψ , (2) where Ψ, C1 and C2 are a function of R,G,B and fcs is defined by the color transform considered. In most cases, the luminance is defined by a weighted average of R,G,B color channels [3,4,5,7,8,9,11], except for Sobol’s method [10]. Then, the final RGB image is obtained either by converting the luminance/chrominance image back to RGB (3) or by using a scaling technique where the ratio of the initial luminance and the treated luminance multiplies the three color channel (4). The particular case of Sobol’s method adds the treated luminance to the log-encoded RGB image (5). { } { }) , , ( , , 2 1 1 C C f B G R new cs new Ψ = − (3) { } { } B G R B G R new new , , , , ⋅ Ψ Ψ = (4) { } { } ) log( ), log( ), log( , , B G R B G R new new + Ψ = (5) Here “·” and “+” are component per component operations. In this article, we study the effect of different luminance definitions on the rendered image for the case of MSR [3,4] and Retinex-based adaptive filtering [5,6]. Our aim is to investigate the relationship between the decorrelation of luminance and chrominance information and the correct rendition of the color after a local tone mapping was applied to the luminance. A measure of correlation The four color transforms that we chose for our tests are described in Table 1. Each of them transforms the linear RGB input image in a luminance/chrominance encoding (2). The first one, “fRGB” transform, simply defines the green channel G as being the luminance and the red and blue channels R,B as being the chrominance. With this transform, the luminance is strongly correlated with the chrominance. The second transform is “fYUV”, which is a linear transform widely used for video processing [14]. The third one, “fLab”, is used for perceptual experiment and is not a linear transformation. The last one “fPCA” is an image-dependent, linear transform based on a PCA applied to the input image, which guarantees perfect decorrelation between components. The luminance is defined by the first principal component. Table 1. The four color transforms tested. fcs Luminance Chrominance Transform RGB Linear G R,B fRGB YUV Linear Y u,v fYUV